TITLE OF THE INVENTION 

APPARATUS AND METHOD FOR REMOTE -CONTROLLING IMAGE 
SENSING APPARATUS IN IMAGE SENSING SYSTEM 

5 

BACKGROUND OF THE INVENTION 

The present invention relates to control method and 
control apparatus for an image sensing system in which 
10 image signals sensed by an image sensing apparatus are 

distributed to clients via network, and a storage medium 
storing program codes for realizing the method. 

An image transmission system in which media data, 
such as an image and sound obtained by a video camera on 
15 a network, such as the Internet, are transmitted to a 

plurality of clients via the network has been developed. 

On a remote monitoring system in which a camera is 
connected to a network, panning, tilting, and zooming 
operation of the camera can be remote-controlled, and an 
20 image obtained by the camera is transmitted to and 

browsed by a plurality of clients via the network is 
available. Such a system is disclosed in Japanese Patent 
Application Laid-Open No. 10-42279. 

In this type of transmission system, however, since 
25 the camera is controlled from a remote location, it is 
difficult for clients to know where the camera is 
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located and which direction the camera is facing. 
Therefore, it is also difficult for the clients to 
browse desired images quickly. 

5 SUMMARY OF THE INVENTION 

The present invention has been made in 
consideration of the above situation, and has as its 
object to provide a method and apparatus for controlling 

10 an image sensing apparatus in an image sensing system 
which enable a client to easily control the image 
sensing apparatus from a remote location. 

According to the present invention, the foregoing 
object is attained by providing a remote control 

15 apparatus for remote controlling an image sensing 

apparatus by changing image sensing conditions of the 
image sensing apparatus, the remote control apparatus 
comprising: map display means for displaying map 
information; state display means for obtaining 

20 parameters of the image sensing apparatus and displaying 
a position and state of the image sensing apparatus on 
the map information displayed by the map display means 
on the basis of the parameters; designation means for 
designating an image sensing area to be sensed by the 

25 image sensing apparatus on the map information; and 
control value calculation means for calculating a 
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control value for controlling the image sensing 
apparatus on the basis of the image sensing area 
designated by the designation means. 

According to the present invention, the foregoing 
5 object is also attained by providing an image sensing 
system which remote controls an image sensing apparatus 
by changing image sensing conditions of the image 
sensing apparatus, the system comprising: map display 
n means for displaying map information; state display 

n 10 means for obtaining parameters of the image sensing 

»a apparatus and displaying a position and state of the 

image sensing apparatus on the map information displayed 
by the map display means on the basis of the parameters; 
^ designation means for designating an image sensing area 

|y 15 to be sensed by the image sensing apparatus on the map 

Q information; and control means for controlling the image 

sensing apparatus on the basis of the designation by the 
designation means. 

Further, the foregoing object is also attained by 
20 providing a remote control method for remote controlling 
an image sensing apparatus by changing image sensing 
conditions of the image sensing apparatus, the method 
comprising: a map display step of displaying map 
information; a parameter obtaining step of obtaining 
25 parameters of the image sensing apparatus; a state 

display step of displaying a position and state of the 
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image sensing apparatus on the map information displayed 
in the map display step on the basis of the parameters 
obtained in the parameter obtaining step; a designation 
step of designating an image sensing area to be sensed 
5 by the image sensing apparatus on the map information; 
and a control step of controlling the image sensing 
apparatus on the basis of the designation in the 
designation step. 

n Furthermore, the foregoing object is also attained 

{a 10 by providing a computer program product comprising a 

Vi computer usable medium having computer readable program 

code means embodied in the medium for remote controlling 
an image sensing apparatus by changing image sensing 
C3 conditions of the image sensing apparatus, the product 

|ij 15 comprising: first computer readable program code means 

y of a map display step for displaying map information; 

second computer readable program code means of a 
parameter obtaining step for obtaining parameters of the 
image sensing apparatus; third computer readable program 
20 code means of a state display step for displaying a 

position and state of the image sensing apparatus on the 
map information displayed in the map display step on the 
basis of the parameters obtained in the parameter 
obtaining step; fourth computer readable program code 
25 means of a designation step for designating an image 

sensing area to be sensed by the image sensing apparatus 
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on the map information; and fifth computer readable 
program code means of control step for controlling the 
image sensing apparatus on the basis of the designation 
in the designation step. 
5 According to the configurations as described above, 

a current position and image sensing states of a camera 
are shown by a camera icon in an easily recognizable 
manner. Furthermore, by directly designating a desired 
area on a graphical user interface, the direction and 

10 zoom value of the camera are easily controlled. 

Other features and advantages of the present 
invention will be apparent from the following 
description taken in conjunction with the accompanying 
drawings, in which like reference characters designate 

15 the same or similar parts throughout the figures thereof. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The accompanying drawings, which are incorporated 
20 in and constitute a part of the specification, 

illustrate an embodiment of the invention and, together 
with the description, serve to explain the principles of 
the invention. 

Fig. 1 is a block diagram illustrating a 
25 configuration of an image sensing system according to an 
embodiment of the present invention; 
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Fig. 2 is a view of a user interface of the present 
invention; 

Figs . 3A to 3D are views for explaining states 
which camera icons show; 
5 Fig. 4 is a flowchart of a display sequence of a 

camera icon; 

Fig. 5 is a flowchart for explaining an operation 
of a map controller; 

Fig. 6 is an explanatory view for explaining a 
10 sequence for obtaining panning and zooming values; 

Figs. 7A to 7C show images of icons before and 
after image sensing areas are designated; and 

Fig. 8 is a memory map of program codes used for 
remote-controlling an image sensing apparatus of the 
15 present invention. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

A preferred embodiment of the present invention 
20 will be described in detail below in accordance with the 
accompanying drawings . 

Fig. 1 is a block diagram illustrating a 
configuration of an image sensing system according to 
the embodiment of the present invention. In Fig. 1, 
25 reference numeral 101 denotes an image sensing apparatus, 
and a camera is used in the embodiment. Further, 
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reference numeral 102 denotes a camera server which is 
realized on the basis of a personal computer and 
connects to the camera 101- The camera server 102 
includes a camera control server 103 and an image server 
5 104, and controls the camera 101 by the camera control 
server 103 . The image server 104 captures an image 
sensed by the camera 101. The camera server 102 can be 
used on a network, and in such a case, the camera server 
p « 102 receives parameters for controlling the camera 101 

^ 10 from an external computer on the network, for instance, 

^ and the camera control server 103 controls the camera 

if rl 

58 101 using the received parameters. Further, a captured 

lis 

if" 

*5 image is transmitted from the image server 104 to an 

C3 external computer via the network. A detailed 

y 15 explanation of the camera server 102 is in Japanese 

l*\ Patent Application Laid-Open No. 10-42279. 

Further, reference numeral 105 denotes a client 
machine, and realized on the basis of a personal 
computer. In the client machine 105, reference numeral 
20 106 denotes a CPU; 107, ROM; 108, a communication 

interface; 109, an input/output interface; and 112, RAM. 
In the RAM 112, a program for realizing the embodiment 
and data necessary for executing the program are stored. 
More specifically, a map controller 113 and objects of 
25 first to N-th camera icons 114-1 to 114-N are stored. 

Further, reference numeral 110 denotes a display device; 



-7- 



and 111, an operation unit, such as a mouse and a 
keyboard. Reference numeral 115 denotes an external 
storage unit for driving an external storage medium, 
such as a floppy disk and a CD-ROM. When the control 
program is stored in the external storage medium instead 
of the RAM 112, the program is read and loaded down from 
the external storage medium. 

Fig. 2 shows a graphical user interface (GUI) 
.according to the embodiment of the present invention. 
^\ >/Vhe upper window is a camera viewer where an image 
iransnktted by the camera server 102 is displayed. 
FurtheA, it is possible to designate the direction, 
panning A and tiling of the camera by operating scroll 
bars arranged on the right and bottom of the transmitted 
15 image. A Camera viewer of this type is explained in 

detail in Japanese Patent Application Laid-Open No. 10- 
42279. 

A map guide window and camera icons are displayed 
below the camera viewer. These images are displayed on 

20 the basis of the map controller 113 and the objects of 
the first to N-th camera icons 114-1 to 114-N stored in 
the RAM 112 . The map guide window is for receiving 
inputs from a mouse operated by a user and controlling 
the camera. The camera icons are for graphically showing 

25 current states (e.g., panning, zooming) of cameras on a 
map of the map guide window. 
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Figs. 3A to 3D are explanatory views for explaining 
states shown by camera icons. In Figs. 3A to 3D, the 
visual fields of cameras are expressed by sector forms. 
Specifically, Fig. 3A shows a zoomed-out state, and Fig. 
5 3B shows a zoomed-in state. 

As shown in Figs. 3A and 3B, by graphically 
displaying an area where the camera is viewing in sector 
forms, a user can easily understand the information, 
such as the area a camera is viewing and the direction 
10 the camera is facing. 

In the embodiment, it is assumed that a plurality 
of clients may simultaneously access the camera (i.e., 
camera server 102), and in such a case, difficulties 
will occur if several clients try to control the camera 
15 at the same time. Therefore, in the embodiment, a single 
client authorized to control the camera (referred to as 
"authorization" hereinafter) is allowed to control the 
camera at a time. 

For example, when a user of a client machine wants 
20 to control the camera, the client machine requests the 
authorization to the camera server 102, and, if no 
client holds the authorization at the time of the 
request, then the authorization is issued to the user. 
The authorization is valid for a predetermined period, 
25 and the user is allowed to control the camera for the 

predetermined period. Whereas, if another client holds 
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the authorization at the time of the request, the 
authorization is issued to the requested client after 
the authorization issued to the other client is expired. 
A period of validation and a priority order of clients 
5 that the authorization is given, for instance, may be 
set differently. Detailed explanation on authorization 
is also disclosed in Japanese Patent Application Laid- 
Open No. 10-42279. 

In the embodiment of the. present invention, when 

10 authorization to control a camera is issued to a client, 
a camera icon corresponding to the camera is surrounded 
by a color frame which is easily distinguished from the 
background image, e.g., a map. Fig. 3C shows a state 
that authorization to control the camera is not issued 

15 to a client, and Fig. 3D shows a state that 

authorization is issued to a client. It should be noted 
that a method for showing that authorization to control 
a camera is issued to a client is not limited to 
surrounding a corresponding camera icon by a color frame, 

20 and any methods may be used as far as whether or not the 
authorization is issued to a client is informed to a 
user. For instance, when authorization is not issued, a 
corresponding camera icon may be shown by dot lines, 
whereas when authorization is issued, the corresponding 

25 camera icon may be shown by solid lines. As another 
example, a color of a camera icon may be changed 



-10- 



depending upon whether or not the authorization to 
control a camera is issued. 

Next, a display sequence of a camera icon is 
explained with reference to Fig. 4. 
5 First in step S401, initialization is performed. 

More specifically, icon images are generated and 
variables are initialized in step S401. Thereafter, the 
client connects to the camera control server 103 in step 
S402, then the process proceeds to step S403. 

10 In step S403, whether or not the connection between 

the client and the camera control server 103 is 
disconnected is checked. If it is, then the process 
proceeds to step S4 04 where the process is completed. 
Whereas, if the connection is still valid, then the 

15 process proceeds to step S405 where the client inquires 
the camera control server 103 of current panning and 
zooming values, and in step S406, the camera icon is 
updated on the basis of the obtained panning and zooming 
values and displayed on the map in the map guide window 

20 shown in Fig . 2 . 

Next, an operation of the map controller 113 is 
explained with reference to Fig. 5. 

First in step S501, initialization is performed. 
More specifically, a map image is generated, and various 

25 editorial functions are initialized. Thereafter, in step 
S502, the map controller 113 connects to the camera 
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server 102 and generates camera icon/icons in step S503. 
The generated camera icon/ icons is /are displayed on the 
map in the map guide window in accordance with a 
flowchart shown in Fig. 4. 



clicked by a user is checked. If it is, the process 
proceeds to step S505, whereas if it is not, the process 
proceeds to step S508. 

In step S505, whether or not a pointer is on a 

10 camera icon when the mouse is clicked is checked. If the 
pointer is on the camera icon (YES in step S505), then 
the process proceeds to step S506 where authorization to 
control a corresponding camera is requested to the 
camera control server 103. Whereas, if the pointer is 

15 not on the camera icon (NO in step S505), then the 

process proceeds to step S507 where vectors for keeping 
the mouse locus, which will be explained later, is 
initialized. 



20 performed by the mouse is checked. If it is, then 
coordinates of a pointer of the mouse are stored as 
vectors of the mouse locus at a predetermined interval, 
and the mouse locus is displayed on the map in the map 
guide window. Examples of displayed mouse loci are 

25 explained later with reference to Figs. 7A to 7C . 



5 



Next, in step S504, whether or not a mouse is 



In step S508, whether or not drag operation is 
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Whereas, if drag operation is not performed, then the 
process proceeds to step S510. 

In step S510, whether or not the mouse is released 
(a button of a mouse which has been held by a user is 
5 released) is checked. If it is not, the process proceeds 
to step S515, whereas if it is, the process proceeds to 
step S511. In step S511, whether or not the client 
machine holds authorization is checked, if it does not, 
the process proceeds to step S515. 

10 If the client machine holds the authorization, then 

in step S512, the minimum value, the maximum value and 
the center value of the coordinate data stored as the 
vectors of mouse locus are obtained for both X and Y 
axes. From these values, the smallest rectangular area 

15 which includes the mouse locus is obtained. 

In step S513, coordinates of the rectangular area 
obtained in step S512 are compared with coordinates of 
the camera icon, and panning and zooming values of the 
camera are determined. 

20 An example of processes performed in steps S512 and 

S513 are explained with reference to Fig. 6. 

First, on the basis of the difference between the 
coordinates of the camera icon and the coordinates of 
the upper left corner of the rectangular area, an angle 

25 91 between the left limit of the angle of view and the 
direction to which the camera is currently facing is 
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obtained. Further, on the basis of the difference 
between the coordinates of the camera icon and the 
coordinates of the lower right corner of the rectangular 
area, an angle 02 between the right limit of the angle 
5 of view and the direction to which the camera is 

currently facing is obtained. Furthermore, on the basis 
of the difference between the coordinates of the camera 
icon and the coordinates of the center of the 
rectangular area, an angle 93 between the direction to 
10 which the camera is going to face and the direction to 
which the camera is currently facing is obtained. 

The aforesaid sequence is realized by executing a 
program stored in the RAM 112 by the CPU 106. 

Further, in step S514, the camera is controlled on 
15 the ba:sis of the values obtained in step S513 in the 

aforesaid manner. Specifically, the client requests the 
camera control server 103 to rotate the direction faced 
by the camera by 03 degrees, and set the angle of view 
(i.e., zoom) of the camera to (01 - 92) degrees. 
20 Thereafter, the vectors of the mouse locus are cleared, 
and the process proceeds to step S515. 

In step S515, the map as a background image, a 
camera icon/ icons, and a mouse locus (if the vectors of 
the mouse locus exist, in other words, when the process 
25 proceeds from step S509 or S510 to step S515) are 
rendered in the aforesaid order. 
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In step S516, whether or not the connection between 
the map controller 113 and the camera control server 103 
is disconnected is checked. If it is, the process 
proceeds to step S517 where the processing is completed. 
5 Whereas, if it is not, the process returns to step S504 
and the steps S504 to S516 are repeated. 

By operating as described above, it is possible to 
display the state of a camera at real time by means of a 
camera icon. 

10 Note, in Fig. 6, a case where the camera icon faces 

to the right is explained for the sake of descriptive 
convenience, however, the direction which the camera 
faces varies in practice. Thus, it is necessary to 
calculate angles in consideration of the direction which 

15 the camera faces to. 

Next, examples of images 1 displayed in the map guide 
window when the client machine is controlled on the 
basis of the flowcharts shown in Figs. 4 and 5 are 
explained with reference to Figs. 7A to 7C. 

20 When the camera icon is originally as shown on the 

left side of Fig. 7A and an upper right area of the 
"lobby" is designated, the panning and zooming of the 
camera are controlled so as to view the designated area 
and the controlled panning and zooming states of the 

25 camera as shown on the right side of Fig. 7A are 
displayed in the map guide window. 
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When the camera icon is originally as shown on the 
left upper portion or left lower portion of Fig. 7B, and 
the angle of view is to be narrowed, or zoom- in 
operation is desired, by designating a smaller area, the 
5 camera zooms in so as to view the designated area and 
the controlled zooming state of the camera as shown on 
the right side of Fig. 7B is displayed in the map guide 
window . 

When the camera icon is originally as shown in the 
10 left upper portion or left lower portion of Fig. 7C, and 
the angle of view is to be widened, or zoom-out 
operation is desired, by designating a wider area, the 
camera zooms out so as to view the designated area and 
the controlled zooming state of the camera as shown on 
15 the right side of Fig. 7C is displayed in the map guide 
window. 

It should be noted that a case where a field of 
vision desired to be sensed by a camera is designated by 
operating the mouse is explained in the above 

20 embodiment; however, the present invention is not 

limited to this, and any method is possible as long as a 
desired area can be designated. For example, a touch 
sensor may be provided on the display so that the 
desired area to be viewed can be designated. 

25 According to the embodiment as described above, a 

current position and image sensing states of a camera 
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are displayed in an easily recognizable manner. 
Furthermore, by directly designating a desired area on 
the GUI, the direction and zoom value of the camera are 
easily controlled. 
5 Further, the object of the present invention can 

also be achieved by providing a storage medium storing 
program codes for performing the aforesaid processes to 
a computer system or apparatus (e.g., a personal 
computer) , reading the program codes, by a CPU or MPU of 

10 the computer system or apparatus, from the storage 
medium, then executing the program. 

In this case, the program codes read from the 
storage medium realize the functions according to the 
embodiment, and the storage medium storing the program 

15 codes constitutes the invention. 

Further, the storage medium, such as a floppy disk, 
a hard disk, an optical disk, a magneto-optical disk, 
CD-ROM, CD-R, a magnetic tape, a non-volatile type 
memory card, and ROM can be used for providing the 

20 program codes . 

Furthermore, besides the aforesaid functions 
according to the above embodiment are realized by 
executing the program codes which are read by a computer, 
the present invention includes a case where an OS 

25 (operating system) or the like working on the computer 
performs a part or the entire processes in accordance 
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with designations of the program codes and realize 
functions according to the above embodiment. 

Furthermore, the present invention also includes a 
case where, after the program codes read from the 
5 storage medium are written in a function expansion card 
which is inserted into the computer or in a memory 
provided in a function expansion unit which is connected 
to the computer, CPU or the like contained in the 
function expansion card or unit performs a part or the 
10 entire process in accordance with designations of the 
program codes and realizes functions of the above 
embodiment . 

In a case where the present invention is applied to 
the aforesaid storage medium, the storage medium stores 

15 program codes corresponding to the flowcharts described 
in the embodiment. Briefly, the storage medium stores 
each module shown as an example of a memory map in Fig. 
8. More specifically, program codes which correspond to 
map display module, parameter obtaining module, camera 

20 icon rendering module, image sensing area input module, 
and image sensing apparatus control module, at least, 
are to be stored in the storage medium. 

The present invention is not limited to the above 
embodiments and various changes and modifications can be 

25 made within the spirit and scope of the present 
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invention. Therefore to apprise the public of the scope 
of the present invention, the following claims are made. 
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